An  Architecture  to  Support  Teleoperation  and  Autonomy 


Ronald  Lumia  and  James  S,  Albus 
Robot  Systems  Division 
National  Bureau  of  Standards 
Gaithersburg,  MD  20899 


"i/niEcopy 

DO  NOT  Rtfl 


The  Flight  Telerobotic  Servicer  (FTS)  will  support  assembly  and 
maintenance  activities  for  the  Space  Station.  In  order  for  the 
FTS  to  evolve  with  technology,  careful  attention  must  be  paid  to 
the  system’s  functional  architecture.  This  paper  describes  an 
approach  to  the  functional  architecture  so  that  teleoperation, 
slated  for  beginning  of  the  program,  and  autonomy,  scheduled 
later  in  the  program,  can  both  be  supported.  The  system  is 
hierarchically  organized  where  task  decomposition,  world 
modeling,  and  sensory  processing  are  explicitly  represented. 
Goals  at  each  level  of  the  hierarchy  are  decomposed  spatially  and 
temporally  into  simpler  tasks  which  become  goals  for  lower 
levels.  The  spatial  decomposition  facilitates  control  and 
coordination  of  multi-  arm  robots. 


1.  INTRODUCTION 

NASA  has  embarked  on  a serious  program  of  research  and 
development  in  anticipation  of  the  robotics  requirements  for  the 
Space  Station  [1].  Robot  related  research  is  currently  in 
progress  at  many  centers  [2-6]  such  as  Langley,  Oak  Ridge 
National  Labs,  the  Jet  Propulsion  Laboratory,  Ames  Research 
Center,  Johnson  Space  Center,  etc.  Since  the  FTS  is  targeted  for 
use  in  the  assembly  and  maintenance  of  the  Space  Station,  the 
envisioned  FTS  will  need  multiple  manipulators,  vision  and  other 
sensory  processing,  world  modeling,  planning,  etc.,  in  order  to 
adequately  perform  its  functions. 

The  NASA  program  for  the  development  of  the  FTS  expects  to  use 
teleoperation  for  the  short  term  with  autonomy  blending  into  the 
control  structure  gradually  as  technology  advances.  In  this  way, 
the  probability  of  success  for  the  FTS  is  enhanced.  This 
presents  certain  architectural  pro.blems,  however.  If  the  FTS  is 
to  evolve  from  teleoperation  toward  autonomy,  the  control  system 
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architecture  must  be  able  to  support  the  transition. 

Evolving  primarily  from  work  done  on  automated  factories  [7],  NBS 
has  developed  a hierarchically  organized  control  system.  The 
NASA/NBS  Standard  Reference  Model  for  Telerobot  Control  System 
Architecture  (NASREM)  [8]  has  been  adopted  by  NASA  for  use  as 
the  model  for  the  FTS  control  system.  This  architecture,  which 
is  actually  comprised  of  the  three  hierarchies  of  task 
decomposition,  world  modeling,  and  sensory  processing,  supports 
the  spectrun  of  control  from  total  teleoperation  to  total 
autonomy . 

The  NASREM  architecture  is  presented  in  this  paper.  It  is  shown 
how  multiple  robot  arm  control  and  coordination  is  supported  by 
illustrating  the  interactions  required  between  task  decomposition 
and  the  world  model  for  two  specific  levels  of  the  NASREM 
hierarchy. 


2.  NASREM  ARCHITECTURE 

The  FTS  will  begin  with  teleoperator  control  where  a human  is  an 
integral  part  of  the  control  loop.  Eventually,  the  mode  of 
operation  will  become  autonomous  where  the  human  gives  the  robot 
commands  to  be  executed  and  the  robot  reports  back  when  the  task 
is  completed.  In  order  to  start  with  teleoperated  control  and 
evolve  toward  autonomous  control  without  a complete  redesign  of 
the  robot  control  system,  serious  thought  must  be  given  to  the 
control  architecture  to  be  sure  that  the  system  has  the  ability 
to  be  easily  modified  as  technological  advances  occur. 

The  NASREM  functional  architecture  for  the  control  system  is 
shown  in  Figure  1.  The  control  system  architecture  is  actually 
composed  of  three  hierarchies:  task  decomposition,  world 
modeling,  and  sensory  processing.  The  task  decomposition 
hierarchy  modules  perform  real-time  planning  and  task  monitoring 
functions.  They  decompose  task  goals  in  terms  of  both  space  and 
time.  The  sensory  processing  hierarchy  supplies  information 
about  the  environment  to  the  world  model.  This  involves  the 
processing  of  sensory  data  so  that  patterns,  features,  events, 
etc.,  can  be  measured  about  the  external  world.  The  modules  of 
the  world  model  perform  two  functions.  First,  the  world  model 
contains  the  best  estimate  of  the  state  of  the  external  world. 
This  can  be  used  to  answer  queries,  make  predictions,  and  reason 
about  the  objects  in  the  world.  Second,  the  world  model  acts  as 
the  interface  between  the  task  decomposition  and  sensory 
processing  hierarchies.  This  promotes  greater  modularity  both  in 
function  and  implementation.  For  example,  in  the  execution  of  a 
particular  goal,  the  task  decomposition  module  may  request  the 
location  of  a certain  object  in  the  environment  from  the  world 
model.  The  best  estimate  of  the  object  location  is  returned 
immediately.  The  task  decomposition  neither  knows  nor  cares 
which  sensors  were  used  determine  the  object  location.  It  only 
matters  that  the  best  estimate  is  returned  with  minimal  time 


2 


A*'  ‘ 


...IK;,;;-? 


.>1 


\- 


...;70  y,. 

-..  rj  ' 

‘V-, ><vO  •’' 

.'S  a -iti 


ftdJ  zo'l  t-.= 


^.■ 


;3 1 ^ t;  iqc  ■'  ,^415 

' v>  di  ^ 

■:'%.‘T^.'V'|i 


m ' 'xc 


ov 


as 


- rti'  - 


It'  , 


-*»V  jv 

/ - 


'I  W'  2T%.\ 

= -i ' 


SQ  ^ flit,  " ■'■  ^ ' ■- , :;  -'-.f^g^  ' Q 


•■Ti^  *.■ 


i -^  '■  . ■..■U'y>-  r^ji  - v^..  ^ fa'5  j 40^;  ifOOtOTf 

'»■■■''  i<;j:.is  ,';.  W-*  ivQ  ^ O.--'  ;X0^^ 


:*.  ^»;1S,'' 
, “i  .■  .V  . - 


.'I-  *i 

■x*i  J :; 


, ’^< ' V, 


<'?  "T  ■ ■ .i'.-  ■-■:■  ..yUO^CT 


V ; '-.Wio  ;rA7  ■:x,, 

■y - ■ i yfc;r^:,,  rri:^  •:'a 

»vr> ; V/ ; v t'iS.Vq;'.  . oa  ■ . atife : » , '■  •.f:/,- 

. '-b" 

____  ef:( ';  7: ; ':  -j’r.sir'f  ■*  ;: 


,:;■:  f bi'  'ri 


v'.i:  >■•'.  -^-'i.  .:••--  ^.■.'  - V 

.,;  rx  'h  ■-  ■ - ■ .-  : 

P,  .'■  ■ "■  ' •'  ■ • ' • '■■ ' ■•  • t' 

:'■  < •■--;!  ;„ 

£3j'T^'v^  i4-'!J  ;.)>:■!.  ':^r.:,i,:n7.:\':....-^^^^/^  » 

-rrv'i^^^  ' I8,i  ' nt).::j:g,z^i.  * ■ ■ 

ttm  < ;.;v.y:V;4  '.' 

-V « r;: : . 1)&' 

, ' . ;'■'  ^4:-  ■ ' " ' ‘ 


;?1  fp^ 

■ 

■■  .:r- 

OfSt, 

_ o©  4 1^4^" 

o.^  ..ti'-f^'  Pid 


-:•  nlfly 
I 5.-# 

'■  ,v 

' -■'  ?'  -I  nm 


'-?  e J t'« 


;.» •^o  ,*  .’‘rr  • 


i'  '7L 


tT54>‘ 


7 


,7',|'.; 

.‘i-  > 


i:4' 


^fi:. 


, A ;■  ' ..'  >'  ,,,,„';  ■■  '',;'7,;,'»-  I'g-'y  ''  '' 


delay. 

2.1.  Task  Decomposition  Hierarchy 

The  task  decomposition  modules  plan  and  execute  the  decomposition 
of  high  level  goals  into  low  level  actions.  Task  decomposition 
involves  both  a temporal  decomposition  where  the  goal  is  broken 
up  into  a sequence  of  actions  along  the  time  line  and  a spatial 
decomposition  where  concurrent  actions  are  executed  by  different 
subsystems.  Each  task  decomposition  module  at  each  level  of  the 
hierarchy  consists  of  a job  assignment  manager  JA,  a set  of 
planners  PL(i),  and  a set  of  executors  EX(i).  These  decompose  the 
input  task  into  both  spatially  and  temporally  distinct  subtasks 
as  shown  in  Figure  2. 

2.2.  World  Modeling  Hierarchy 

The  world  modeling  modules  model  and  evaluate  the  state  of  the 
world.  The  world  model  is  the  system's  best  estimate  and 
evaluation  of  the  history,  current  state,  and  possible  future 
states  of  the  world,  including  the  states  of  the  system  being 
controlled.  The  world  model,  as  shown  in  Figure  3,  performs  the 
following  functions: 

1.  Maintains  the  data  in  the  world  model  by  accepting 
information  from  the  sensory  system.  This  keeps  the 
model  of  the  world  in  registration  with  the  physical 
world. 

2.  Provides  predictions  of  expected  sensory  input  to  the 
corresponding  sensory  processing  modules  based  on  the 
state  of  the  task  and  estimates  of  the  external  world. 

3.  Answers  "What  if?"  questions  asked  by  the  planners  in 
the  corresponding  level  task  decomposition  modules. 
The  world  modeling  modules  evaluate  the  results  of 
hypothesized  actions. 

4.  Answers  "What  is?"  questions  asked  by  the  executors 
in  the  corresponding  level  task  decomposition  modules. 
The  task  executor  can  request  the  values  of  any  system 
variable. 


2.3.  Sensory  Processing  Hierarchy 

The  sensory  processing  hierarchy  modules  recognizes  patterns, 
detects  events,  filters  and  integrates  sensory  information  over 
space  and  time,  and  reports  this  information  to  the  world  model 
to  keep  it  in  registration  with  the  external  world.  At  each 
level,  sensory  processing  modules  compare  world  model  predictions 
with  sensory  observations  and  compute  correlation  and  difference 
functions.  These  are  integrated  over  time  and  space  so  as  to 
fuse  sensory  information  from  multiple  sources  over  extended  time 
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intervals  as  shown  in  Figure  4, 

2.4.  Operator  Interface 

The  control  architecture  supports  an  operator  interface  at  each 
level  in  the  hierarchy.  The  operator  interface  provides  a means 
by  which  human  operators,  either  in  the  space  station  or  on  the 
ground,  can  control , observe,  or  supervise  the  telerobot.  Each 
level  of  the  task  decomposition  hierarchy  provides  an  interface 
where  the  human  operator  can  assume  control.  The  task  commands 
into  any  level  can  be  derived  either  from  the  higher  level  task 
decomposition  module,  from  the  operator  interface,  or  from  some 
combination  of  each.  Using  a variety  of  input  devices  such  as  a 
joystick,  mouse,  trackball,  light  pen,  keyboard,  voice  input, 
etc.,  a human  operator  can  enter  the  control  hierarchy  at  any 
level  at  any  time  of  his  choosing  to  monitor  a process,  to  insert 
information,  to  interrupt  automatic  operation  and  take  control  of 
the  task  being  performed,  or  to  apply  human  intelligence  to 
sensory  processing  or  world  modeling  functions.  Table  1 
illustrates  the  types  of  interaction  an  operator  may  have  at  each 
level . 

3.  MULTI -ARM  CONTROL 

Multiple  manipulators  have  been  used  for  many  years  time  in 
teleoperated  mode  for  such  applications  in  the  nuclear  industry. 
Originally,  the  masters  and  slaves  were  coupled  mechanically  but 
technology  now  supports  an  electronic  interconnection.  The 
system  is  able  to  remain  stable  in  teleoperated  mode  because  the 
operator  has  some  force  feedback  from  the  manipulator. 
Traditionally,  this  has  been  done  by  force  reflection  which  can 
be  implemented  in  several  ways  [9].  It  becomes  quite  challenging 
to  control  multiple  robots  autonomously  rather  than  by 
teleoperation. 

Freund  [10]  considered  the  problem  of  two  independent  robots 
working  in  the  same  workspace.  His  work  was  mainly  concerned 
with  avoiding  collision  between  independent  robots.  He  used  a 
hierarchically  organized  nonlinear  control  technique  with  an 
accurate  model  of  the  robot  dynamics  to  plan  the  trajectories  for 
both  robots  simultaneously.  While  collision  avoidance  is 
certainly  important,  his  work  did  not  address  the  problem  of 
cooperation  between  two  robots  executing  a task. 

The  automatic  control  of  coordinated  multiple  manipulators 
presents  a subtle  difficulty  because  even  with  relatively  small 
position  errors,  very  large  forces  can  be  generated  when  closed 
kinematic  chains  are  formed.  Luh  [11]  suggested  an  approach 
where  one  robot  acts  as  the  leader  while  the  other  robot  acts  as 
a follower.  The  desired  motion  of  the  leader  is  planned  based  on 
the  desired  motion  of  the  object.  Given  the  state  variables  of 
the  leader,  which  include  joint  positions,  velocities,  forces, 
etc.,  the  holonomic  constraints  on  the  position  and  orientation 
of  the  follower  can  be  calculated  in  real  time  and  used  for 
control . 
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An  approach  to  multiple  arm  coordination  can  be  implemented  using 
the  NASREM  archiecture.  Hierarchically  organized  multi-arm 
coordinated  control  starts  at  the  task  level  as  illustrated  in 
Figure  5.  Suppose  that  the  task  is  to  move  object  O to  position 
P.  The  job  assignor  (JA)  for  the  task  level  first  determines 
which  type  of  motion  strategy  is  most  appropriate:  single  arm, 
dual  arm,  etc.,  by  suggesting  to  the  world  model  the  various 
alternatives  and  then  choosing  the  best  evaluation  score.  The 
evaluation  of  a specific  strategy  can  be  based  on  the  weight  of 
the  object,  the  location  of  legal  grip  points,  etc.  The  job 
assigner  also  designates  the  leader  and  follower  arms. 

At  this  point,  the  planners  (PL)  in  the  task  level  for  the  leader 
and  follower  are  accessing  the  world  model  to  determine  which 
gripper  is  required  for  the  part  and  the  precise  location  of  each 
robot's  grasp  point.  The  executors  (EX)  at  this  level  need 
information  about  the  current  gripper  on  the  robot  in  order  to 
send  out  the  proper  sequence  of  E-Moves  (elementary  movements) 
required  to  perform  the  task.  This  decomposition  continues 
through  the  e-move  and  primitive  with  two  parallel  chains  of 
control,  one  for  each  robot.  Each  chain,  however,  has  a slightly 
different  method  of  execution  because  one  robot  has  been 
designated  to  be  the  leader  and  the  other  the  follower. 

At  the  servo  level,  following  Luh's  algorithm,  the  leader  is 
operating  in  the  mode  of  simple  position  control.  The  only 
difference  between  the  leader's  activities  for  coordinated 
activity  and  independent  activity  is  watching  the  world  model  to 
wait  until  the  follower  robot  is  in  the  correct  state.  The 
follower,  however,  operates  quite  differently.  The  follower, 
illustrated  in  Figure  6,  is  constantly  interrogating  the  world 
model  for  its  current  position  and  forces.  The  world  model  also 
is  needed  to  provide  the  position  and  forces  generated  by  the 
leader  robot.  These  are  combined  in  the  motion  control  algorithm 
to  effect  coordinated  movement. 

This  approach  is  only  one  way  in  which  coordinated  robot  control 
can  be  performed.  By  changing  the  algorithms  in  the  servo  level 
or  any  other  level,  different  strategies  for  dual  arm  motion 
control  can  be  compared  and  contrasted  for  different 
applications. 


4.  CONCLUSIONS 

The  use  of  two  arms  in  robot  tasks  opens  up  new  research  areas  in 
space  as  well  as  terrestrial  applications.  A standard  reference 
model  architecture  was  presented  which  supports  the  evolution  of 
robot  control  from  teleoperation  to  autonomy.  Using  the  NASREM 
architecture,  it  was  shown  how  multiple  armed  robots  could  be 
coordinated  in  the  execution  of  the  task  of  free  space  motion. 
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TABLE  1 


OPERATOR  INTERACTION  AT  EACH  LEVEL 


LEVEL 

TYPE  OF  INTERACTION 

At  the  servo 

replica  master,  individual  joint  position,  rate, 
or  force  controllers. 

above  servo 

joy  stick  to  perform  resolved  motion  force/rate 
control 

above  prim 

indicate  safe  motion  pathways.  Robot  computes 
dynamically  efficient  movements 

above  e-move 

graphically  or  symbolically  define  key  poses, 
menus  to  choose  elemental  moves. 

above  task 

specify  tasks  to  be  performed  on  objects. 

above  bay 

reassign  telerobots  to  different  service  bays, 
insert,  modify,  and  monitor  plans  describing 
servicing  task  sequences. 

above  mission 

reconfigure  servicing  mission  priorities. 
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OPERATOR  INTERFACE 
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FIGURE  3’  Rinctlons  Perfonned  by  M Modules  in  the  World  Model 
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